R Notebook

What tweets got the most attention. frequency of tweets posted on a topic

library(rtweet)
library(ggplot2)
library(tidyr)
library(dplyr)

Attaching package: 㤼㸱dplyr㤼㸲

The following objects are masked from 㤼㸱package:stats㤼㸲:

    filter, lag

The following objects are masked from 㤼㸱package:base㤼㸲:

    intersect, setdiff, setequal, union

library(tidytext)
library(stringr)
library(scales)
library(readr)

Attaching package: 㤼㸱readr㤼㸲

The following object is masked from 㤼㸱package:scales㤼㸲:

    col_factor

library(lubridate)

Attaching package: 㤼㸱lubridate㤼㸲

The following objects are masked from 㤼㸱package:base㤼㸲:

    date, intersect, setdiff, union

library(vader)
library(topicmodels)
library(quanteda)

Package version: 3.0.0
Unicode version: 13.0
ICU version: 69.1
Parallel computing: 8 of 8 threads used.
See https://quanteda.io for tutorials and examples.

library(lubridate)
library(zoo)

Attaching package: 㤼㸱zoo㤼㸲

The following object is masked from 㤼㸱package:quanteda㤼㸲:

    index

The following objects are masked from 㤼㸱package:base㤼㸲:

    as.Date, as.Date.numeric

library(plotly)

Registered S3 method overwritten by 'data.table':
  method           from
  print.data.table     
Registered S3 method overwritten by 'htmlwidgets':
  method           from         
  print.htmlwidget tools:rstudio

Attaching package: 㤼㸱plotly㤼㸲

The following object is masked from 㤼㸱package:ggplot2㤼㸲:

    last_plot

The following object is masked from 㤼㸱package:stats㤼㸲:

    filter

The following object is masked from 㤼㸱package:graphics㤼㸲:

    layout

library(forcats)
library(igraph)

Attaching package: 㤼㸱igraph㤼㸲

The following object is masked from 㤼㸱package:plotly㤼㸲:

    groups

The following objects are masked from 㤼㸱package:lubridate㤼㸲:

    %--%, union

The following objects are masked from 㤼㸱package:dplyr㤼㸲:

    as_data_frame, groups, union

The following object is masked from 㤼㸱package:tidyr㤼㸲:

    crossing

The following objects are masked from 㤼㸱package:stats㤼㸲:

    decompose, spectrum

The following object is masked from 㤼㸱package:base㤼㸲:

    union

library(ggraph)
library(widyr)
library(qdapRegex)

Attaching package: 㤼㸱qdapRegex㤼㸲

The following object is masked from 㤼㸱package:dplyr㤼㸲:

    explain

The following object is masked from 㤼㸱package:ggplot2㤼㸲:

    %+%

origop <- options("httr_oauth_cache")
options(httr_oauth_cache = TRUE)

api_key <- "5PgtS7ljq5ZbBoXnemU5qHe62"
api_secret <- "M44LeduQ4zoyDxQIkAFjeIJrpDhWnb5xASDvhahTlrAvOhN7fx"
access_token <- "743029724750942208-JLEp26XrjwvQ1CPJUXwvdUMLka82cgx"
access_secret <- "XRMeMBaOgQy2BC1Bd9iJARfMIyK40VKyII1ZRcf9nS0qd"
token <- create_token(
  app = "KyleResearchApp",
  consumer_key = api_key,
  consumer_secret = api_secret,
  access_token = access_token,
  access_secret = access_secret
)

media_agency_df <- read_csv("data_in/media_agency_tweets.csv")

-- Column specification -----------------------------------------------------------------------------------------------------------
cols(
  .default = col_character(),
  created_at = col_datetime(format = ""),
  display_text_width = col_double(),
  reply_to_status_id = col_logical(),
  reply_to_user_id = col_logical(),
  reply_to_screen_name = col_logical(),
  is_quote = col_logical(),
  is_retweet = col_logical(),
  favorite_count = col_double(),
  retweet_count = col_double(),
  quote_count = col_logical(),
  reply_count = col_logical(),
  symbols = col_logical(),
  ext_media_type = col_logical(),
  quoted_status_id = col_logical(),
  quoted_text = col_logical(),
  quoted_created_at = col_logical(),
  quoted_source = col_logical(),
  quoted_favorite_count = col_logical(),
  quoted_retweet_count = col_logical(),
  quoted_user_id = col_logical()
  # ... with 30 more columns
)
i Use `spec()` for the full column specifications.

207 parsing failures.
 row                  col           expected               actual                              file
2282 reply_to_status_id   1/0/T/F/TRUE/FALSE x1405793506459783168 'data_in/media_agency_tweets.csv'
2282 reply_to_user_id     1/0/T/F/TRUE/FALSE x1358665806121467908 'data_in/media_agency_tweets.csv'
2282 reply_to_screen_name 1/0/T/F/TRUE/FALSE LA89db               'data_in/media_agency_tweets.csv'
3687 reply_to_user_id     1/0/T/F/TRUE/FALSE x240546747           'data_in/media_agency_tweets.csv'
3687 reply_to_screen_name 1/0/T/F/TRUE/FALSE MasegoRahlaga        'data_in/media_agency_tweets.csv'
.... .................... .................. .................... .................................
See problems(...) for more details.

Comparing Tweet Archives

Some tweets fetched date further back. The 3200 tweet pull per user causes this. It tells us that agencies like News24 and SABC News Tweet more daily than Daily Maverick.

ggplot(media_agency_df, aes(x = created_at, fill = screen_name)) +
  geom_histogram(position = "identity", bins = 20, show.legend = FALSE) +
  facet_wrap(~screen_name, ncol = 1)

We first clean the original text to remove links, punctuation, digits, links, @’s. Then we tokenize the Tweets and remove stop words from tidytext and our own stopword dictionary. We also use the “twitter” token to handle any left over @’s and URLS.

#tidy df and unnest
tidy_media_df <- media_agency_df %>% 
  #filter(str_detect(text, fixed(covid_dictionary, ignore_case = TRUE))) %>%
  unnest_tokens(word, text, token = "tweets") %>% 
  filter(!word %in% stop_words$word,
         !word %in% agency_stop_words$word,
        # !word %in% negated_words$word2,
         !word %in% str_remove_all(stop_words$word, "'"),
         str_detect(word, "[a-z]"))

Using `to_lower = TRUE` with `token = 'tweets'` may not preserve URLs.

From our tidyed Tweet dataset, we look for the top words that appear. This will give us a good idea of what topics are being discussed the most. We find that COVID-19 has been the main topic of discussion. President appears second as President Ramaphosa of South Africa usually adresses the nation regarding COVID-19 information. Additionally, Zuma also appears as he is mentioned as “former president Zuma” in most articles. Zuma appears more as his recent court avoidance and sentencing is being Tweeted. General words surrounding the COVID-19 topic as it is still the main pressure on the country, espicially involving Gauteng’s rise in infections.

#top words
top_words <- tidy_media_df %>% 
  anti_join(stop_words) %>% 
  count(word) %>% 
  arrange(desc(n))
top_words %>%
  slice(1:20) %>%
  ggplot(aes(reorder(word, -n), n, fill = word)) +
  geom_bar(stat="identity") +
  theme_minimal() +
  theme(
    axis.text.x = element_text(angle = 60, hjust = 1, size = 13),
    plot.title = element_text(hjust = 0.5, size = 18)
    ) +
  ylab("Frequency") +
  xlab ("") +
  ggtitle("Most frequent media agency tweets") +
  guides(fill=FALSE)

News

#tf-idf
tidy_media_tf_idf <- tidy_media_df %>% 
  select(screen_name, word) %>% 
  count(word, screen_name) %>% 
  bind_tf_idf(word, screen_name, n) 

#calculate a frequency for each agency and word
frequency <- tidy_media_tf_idf %>% 
  group_by(screen_name) %>% 
  count(word, sort = TRUE) %>% 
  left_join(tidy_media_df %>% 
              group_by(screen_name) %>% 
              summarise(total = n())) %>%
  mutate(freq = n/total)

Joining, by = "screen_name"

tidy_media_tf_idf %>% filter(word == "iss")

tidy_media_df %>% filter(word == "iss")

tidy_media_tf_idf %>%
  group_by(screen_name) %>%
  slice_max(tf_idf, n = 15) %>%
  ungroup() %>%
  ggplot(aes(tf_idf, fct_reorder(word, tf_idf), fill = screen_name)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~screen_name, ncol = 2, scales = "free") +
  labs(x = "tf-idf", y = NULL)

By modelling bigrams and trigrams for our dataset gives us a better understanding of what topic is being discussed with each word. We can then also see which sentiments are incorrectly labeled. “not good” gives better context of a negative sentiment, rather than it being incorrectly identified as positive good.

We give more weight to words that appear more often with the incorrect sentiment. We can now reverse the sentiment of these words once VADER has been run on the dataset.

Trigrams were modelled but were not necessary as bigrams provided enough information

#trigrams
tidy_trigram_df <- media_agency_df %>%
  filter(!str_detect(text, "^RT")) %>%
  mutate(text = gsub(" ?(f|ht)tp(s?)://(.*)[.][a-z]+", " ", text)) %>%
  unnest_tokens(trigram, text, token = "ngrams", n = 3) %>%
  separate(trigram, c("word1", "word2", "word3"), sep = " ")

trigrams_filtered <- trigrams_separated %>%
  filter(!word1 %in% stop_words$word) %>%
  filter(!word2 %in% stop_words$word) %>% 
  filter(!word3 %in% stop_words$word) 

# new trigrams counts:
trigram_counts <- trigrams_filtered %>% 
  count(word1, word2, word3, sort = TRUE)

trigrams_united <- trigrams_filtered %>%
  unite(trigram, word1, word2, word3, sep = " ")

#tf-idf trigrams
tidy_trigram_df <- trigrams_united %>%
  count(screen_name, trigram) %>%
  bind_tf_idf(trigram, screen_name, n) %>%
  arrange(desc(tf_idf))

tidy_trigram_df %>%
  group_by(screen_name) %>%
  slice_max(tf_idf, n = 10) %>%
  ungroup() %>%
  ggplot(aes(tf_idf, fct_reorder(trigram, tf_idf), fill = screen_name)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~screen_name, ncol = 2, scales = "free") +
  labs(x = "tf-idf", y = NULL)

#sentiment
ggplotly(ggplot(data=media_vader_df, aes(x=created_at)) +
  geom_line(aes(y=rollmean(compound, k=30, na.pad = TRUE)), color="pink", size=.5)+
  geom_smooth(aes(y=compound)) +
  theme_minimal()+
  facet_wrap(~screen_name) +
  scale_y_continuous(expand = c(0,0)))

`geom_smooth()` using method = 'gam' and formula 'y ~ s(x, bs = "cs")'
Removed 979 rows containing non-finite values (stat_smooth).

install.packages("ldatuning")

WARNING: Rtools is required to build R packages but is not currently installed. Please download and install the appropriate version of Rtools before proceeding:

https://cran.rstudio.com/bin/windows/Rtools/
Installing package into 㤼㸱C:/Users/Administrator/Documents/R/win-library/4.0㤼㸲
(as 㤼㸱lib㤼㸲 is unspecified)
trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.0/ldatuning_1.0.2.zip'
Content type 'application/zip' length 541109 bytes (528 KB)
downloaded 528 KB

package ‘ldatuning’ successfully unpacked and MD5 sums checked

The downloaded binary packages are in
    C:\Users\Administrator\AppData\Local\Temp\RtmpQZiRta\downloaded_packages

install.packages("devtools")

Error in install.packages : Updating loaded packages

devtools::install_github("nikita-moor/ldatuning")

WARNING: Rtools is required to build R packages, but is not currently installed.

Please download and install Rtools 4.0 from https://cran.r-project.org/bin/windows/Rtools/.
Downloading GitHub repo nikita-moor/ldatuning@HEAD

These packages have more recent versions available.
It is recommended to update all of them.
Which would you like to update?

1: All                               
2: CRAN packages only                
3: None                              
4: colorspace (2.0-1 -> 2.0-2) [CRAN]
5: Rcpp       (1.0.6 -> 1.0.7) [CRAN]
6: cli        (2.5.0 -> 3.0.0) [CRAN]

install.packages("devtools")
1

colorspace (2.0-1 -> 2.0-2) [CRAN]
Rcpp       (1.0.6 -> 1.0.7) [CRAN]
cli        (2.5.0 -> 3.0.0) [CRAN]

Installing 3 packages: colorspace, Rcpp, cli
Installing packages into 㤼㸱C:/Users/Administrator/Documents/R/win-library/4.0㤼㸲
(as 㤼㸱lib㤼㸲 is unspecified)

  There is a binary version available but the source version is later:

  Binaries will be installed

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.0/colorspace_2.0-2.zip'
Content type 'application/zip' length 2649327 bytes (2.5 MB)
downloaded 2.5 MB

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.0/Rcpp_1.0.6.zip'
Content type 'application/zip' length 3253582 bytes (3.1 MB)
downloaded 3.1 MB

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.0/cli_3.0.0.zip'
Content type 'application/zip' length 756454 bytes (738 KB)
downloaded 738 KB

package ‘colorspace’ successfully unpacked and MD5 sums checked

cannot remove prior installation of package 㤼㸱colorspace㤼㸲problem copying C:\Users\Administrator\Documents\R\win-library\4.0\00LOCK\colorspace\libs\x64\colorspace.dll to C:\Users\Administrator\Documents\R\win-library\4.0\colorspace\libs\x64\colorspace.dll: Permission deniedrestored 㤼㸱colorspace㤼㸲

package ‘Rcpp’ successfully unpacked and MD5 sums checked

cannot remove prior installation of package 㤼㸱Rcpp㤼㸲problem copying C:\Users\Administrator\Documents\R\win-library\4.0\00LOCK\Rcpp\libs\x64\Rcpp.dll to C:\Users\Administrator\Documents\R\win-library\4.0\Rcpp\libs\x64\Rcpp.dll: Permission deniedrestored 㤼㸱Rcpp㤼㸲

package ‘cli’ successfully unpacked and MD5 sums checked

The downloaded binary packages are in
    C:\Users\Administrator\AppData\Local\Temp\RtmpQZiRta\downloaded_packages

WARNING: Rtools is required to build R packages, but is not currently installed.

Please download and install Rtools 4.0 from https://cran.r-project.org/bin/windows/Rtools/.

  
  
  
√  checking for file 'C:\Users\Administrator\AppData\Local\Temp\RtmpQZiRta\remotesa3c6b60dd8\nikita-moor-ldatuning-1e225c0/DESCRIPTION' (368ms)

  
  
  
-  preparing 'ldatuning':
   checking DESCRIPTION meta-information ...
  
√  checking DESCRIPTION meta-information

  
  
  
-  checking for LF line-endings in source and make files and shell scripts
-  checking for empty or unneeded directories

  
-  building 'ldatuning_1.0.2.tar.gz'

  
   

Installing package into 㤼㸱C:/Users/Administrator/Documents/R/win-library/4.0㤼㸲
(as 㤼㸱lib㤼㸲 is unspecified)

* installing *source* package 'ldatuning' ...
** using staged installation
** R
** demo
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
  converting help for package 'ldatuning'
    finding HTML links ... done
    Arun2010                                html  
Rd warning: C:/Users/Administrator/AppData/Local/Temp/RtmpmMHwap/R.INSTALL3ef4783647b1/ldatuning/man/Arun2010.Rd:10: file link 'LDA' in package 'topicmodels' does not exist and so has been treated as a topic
Rd warning: C:/Users/Administrator/AppData/Local/Temp/RtmpmMHwap/R.INSTALL3ef4783647b1/ldatuning/man/Arun2010.Rd:12: file link 'DocumentTermMatrix' in package 'tm' does not exist and so has been treated as a topic
Rd warning: C:/Users/Administrator/AppData/Local/Temp/RtmpmMHwap/R.INSTALL3ef4783647b1/ldatuning/man/Arun2010.Rd:14: file link 'simple_triplet_matrix' in package 'slam' does not exist and so has been treated as a topic
    CaoJuan2009                             html  
Rd warning: C:/Users/Administrator/AppData/Local/Temp/RtmpmMHwap/R.INSTALL3ef4783647b1/ldatuning/man/CaoJuan2009.Rd:10: file link 'LDA' in package 'topicmodels' does not exist and so has been treated as a topic
    Deveaud2014                             html  
Rd warning: C:/Users/Administrator/AppData/Local/Temp/RtmpmMHwap/R.INSTALL3ef4783647b1/ldatuning/man/Deveaud2014.Rd:10: file link 'LDA' in package 'topicmodels' does not exist and so has been treated as a topic
    FindTopicsNumber                        html  
    finding level-2 HTML links ... done

Rd warning: C:/Users/Administrator/AppData/Local/Temp/RtmpmMHwap/R.INSTALL3ef4783647b1/ldatuning/man/FindTopicsNumber.Rd:20: file link 'DocumentTermMatrix' in package 'tm' does not exist and so has been treated as a topic
Rd warning: C:/Users/Administrator/AppData/Local/Temp/RtmpmMHwap/R.INSTALL3ef4783647b1/ldatuning/man/FindTopicsNumber.Rd:22: file link 'simple_triplet_matrix' in package 'slam' does not exist and so has been treated as a topic
Rd warning: C:/Users/Administrator/AppData/Local/Temp/RtmpmMHwap/R.INSTALL3ef4783647b1/ldatuning/man/FindTopicsNumber.Rd:29: file link 'LDA' in package 'topicmodels' does not exist and so has been treated as a topic
Rd warning: C:/Users/Administrator/AppData/Local/Temp/RtmpmMHwap/R.INSTALL3ef4783647b1/ldatuning/man/FindTopicsNumber.Rd:40: file link 'LDA' in package 'topicmodels' does not exist and so has been treated as a topic
    FindTopicsNumber_plot                   html  
    Griffiths2004                           html  
Rd warning: C:/Users/Administrator/AppData/Local/Temp/RtmpmMHwap/R.INSTALL3ef4783647b1/ldatuning/man/Griffiths2004.Rd:10: file link 'LDA' in package 'topicmodels' does not exist and so has been treated as a topic
    ldatuning                               html  
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (ldatuning)

library("ldatuning")


library("topicmodels")
data("AssociatedPress", package="topicmodels")
dtm <- AssociatedPress[1:10, ]


result <- FindTopicsNumber(
  tidy_matrix,
  topics = seq(from = 2, to = 15, by = 1),
  metrics = c("Griffiths2004", "CaoJuan2009", "Arun2010", "Deveaud2014"),
  method = "Gibbs",
  control = list(seed = 77),
  mc.cores = 2L,
  verbose = TRUE
)

fit models... done.
calculate metrics:
  Griffiths2004... done.
  CaoJuan2009... done.
  Arun2010... done.
  Deveaud2014... done.

FindTopicsNumber_plot(result)

`guides(<scale> = FALSE)` is deprecated. Please use `guides(<scale> = "none")` instead.